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© Method and apparatus for storing and merging multiple optically scanned images. 



© A method for storing multiple scanned images of 
a response-marked form document comprises scan- 
ning a copy of the form document on which no 
response information has been marked, using a 
scanning method that detects substantially all visual 
information present on the form document and pro- 
duces a pixel image thereof and storing that pixel 
image data in a computer storage device. Additional 
steps comprise scanning at least one copy of the 
same form document that has been marked with at 
least one response mark that absorbs light falling 
outside the visible spectrum, using a scanning meth- 

Fig.l 



od that detects only the presence of light outside the 
visible spectrum and that produces a pixel image of 
said at least one response mark, then compressing 
that pixel image to reduce the amount of pixel data 
used to represent areas of the form document bear- 
ing no response marks and storing the compressed 
pixel image data in a computer storage device. The 
visual scan pixel image and the image resulting from 
a scan for response marks can be merged together 
to produce a signal image that is comparable in 
quality and appearance to an image from a single, 
conventional visual-mode scan. 
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RELATED APPLICATION 

This application is related to a co-pending Eu- 
ropean Patent Application filed in the name of 
National Computer Systems, Inc., and concurrently 
with the present application, entitled IMAGE PRO- 
CESSING SYSTEM FOR DOCUMENTARY DATA, 
and which is hereby incorporated by reference in 
the present application. 

TECHNICAL FIELD 

The present invention relates generally to the 
field of optical mark scanning apparatus and image 
scanning and storing apparatus. More particularly, 
the present invention relates to a method and ap- 
paratus for capturing scanned images from multiple 
scanned documents having some redundant data 
and other variable data, efficiently storing scanned 
document images by storing redundant data only 
once and then merging two or more stored images 
to create a single combined image. 

BACKGROUND ART 

Optical mark scanning apparatus (hereinafter 
referred to as optical mark reading or OMR sys- 
tems) are well known in the prior art. There are a 
wide variety of applications that use OMR systems 
and scannable forms in the large-scale gathering of 
discrete pieces of information recorded on docu- 
ments (e.g., scores from standardized tests, census 
information, preference surveys, etc.). OMR sys- 
tems generally sense data recorded with marks 
containing pigment that absorbs light in the near 
infrared range (e.g., No. 2 pencil mark) in specific, 
predefined fields on a form. (This is called NIR- 
rnode scanning.) Such data are position-encoded, 
in that their meaning is interpreted based on where 
the data has been marked on the form document 
used to record it. A primary advantage of NIR- 
mode scanning is that it allows separation of the 
respondent-marked variable data from the standard 
background printing on the same document 

Although a more recent development, visual 
image scanning apparatus (hereinafter referred to 
as VIS mode scanners or VIS systems) are also 
well known in the prior art. Most of the applications 
for current VIS systems have developed out of the 
desire to find replacements for traditional microfilm 
image capture systems. As a result, the focus of 
current VIS systems is on the acquisition of the 
total scanned image of a document as compared 
to the detection, dark mark resolution and process- 
ing of discrete pieces of information recorded in 
predefined fields done by current OMR systems. 

One of the limitations with current OMR sys- 
tems that focus on predefined fields is their lack of 



any ability to scan or reproduce the context of a 
scannable form surrounding the predefined fields. 
As a result, the retrieval and viewing or hard copy 
replication of the stored or archived discrete, raw 

5 information from a form scanned by an OMR sys- 
tem is of limited value, because there is no capa- 
bility for interpreting that information without resort- 
ing to the form, a transparent overlay, or even a 
booklet associated with the form, as is the case in 

io standardized testing. 

VIS systems, on the other hand, are capable of 
reproducing the entire contents of a scannable 
form (both standard background printing and any 
variable data) as a digitized, pixel image, but any 

15 associated OMR processing logic cannot easily 
interpret or analyze any of the discrete, variable 
information contained in a VIS-mode pixel image of 
the form. A great deal of digitized data, both fixed 
and variable, can be collected by VIS images, but 

20 the meaning of the variable data remains to be 
extracted and not all of the VIS-mode data col- 
lected may be of interest. For example, if an 8 1/2" 
X 11" piece of paper is scanned at a resolution of 
120 dots per inch (DPI), vertically and horizontally, 

25 and with an eight-bit grayscale, more than 1.3 
megabytes worth of bit-level scanned information 
may be generated from this single page. Obvi- 
ously, a tremendous amount of computer process- 
ing power may be necessary to extract and ana- 

30 lyze significant bit-level information representing 
the variable data or localized pixel segments (often 
called areas of interest or AOI's) from a form 
scanned with a VIS system. More importantly, the 
storage requirements for storing all of this informa- 

35 tion in a situation where hundreds or thousands of 
documents are being processed to capture a com- 
plete pixel image as a by-product of capturing the 
variable data is so large as to be prohibitively 
expensive in most applications. When data are 

40 recorded on standard forms, much of the data 
collected in a VIS system is redundant What is 
needed is a scanning system and method that 
permits repeated data on standard forms to be 
scanned and stored once, while variable data, such 

45 as pencil marked test answers, signatures, other 
respondent-marked data, are scanned and stored 
at every occurrence and that permits the relation- 
ship between variable data and repeated data to be 
recovered. 

50 

SUMMARY OF THE INVENTION 

In the output presentation of document images 
captured by scanning, where output may be either 
55 to a CRT (or other electronic) display screen or in 
hard-copy reproductions of the originals, there are 
certain advantages resulting from a means to 
merge two or more separately captured images 
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into a common output format. This technique is 
especially useful if the distinct images are captured 
under different, selective scanning conditions, such 
that one or more of the images to be merged 
contains only an incomplete portion of the material 
printed or marked on a source document. 

For example, the first image may consist of a 
"master form", having been generated from a 
visual-mode scan of a blank (no filied-in variable 
data) example of the form document. Such a form 
usually contains a preprinted configuration that in- 
forms the user of data to be recorded and where to 
record it. The second document image, in turn, 
may result from an NIR-mode scan of a copy of 
the same "master form", but in this case contain- 
ing filled-in variable data recorded with an ordinary 
No. 2 pencil. Such a document may be called a 
"response-marked form". The resulting bit-map im- 
age of a response-marked form contains only two 
"classes" of pixels; those comprising the white 
(data void) background of the document, and the 
pixels that have appreciable grayscale content, re- 
presenting areas on the form containing the pencil 
marks. The areas containing the preprinted text, 
lines, graphics or "boiler plate", as this information 
is sometimes called, are included in the white 
background class, because the colored inks used 
to print this matter on the form are not detected in 
the infrared spectrum. (Such inks are sometimes 
referred to as dropout colors, with respect to the 
defined NIR-mode spectral range for a given scan 
capture.) 

Because only the variable, pencil data are de- 
tected by scanning in the NIR-mode (appearing as 
occasional dark marks against a more or less uni- 
form "white background") the potential for efficient 
data compression is increased significantly, as it is 
not necessary for the compression algorithm to 
deal with the redundant data that would otherwise 
consume com pressed -data storage space if 
scanned in the visible (ink-detected) mode. There- 
fore, near infrared scanning of response-marked 
forms-when the variable response data can be 
restricted to No. 2 pencil entries or other marking 
instruments producing detectable marks-is an at- 
tractive means of capturing, compressing and stor- 
ing the variable response data from a batch of 
response-marked forms containing significant data 
that is redundant from form to form, due to the 
standard printing on the master form. 

In accordance with the present invention, a 
method and apparatus for capturing, storing and 
merging multiple scanned images captures OMR 
scanned images of variable data marked on a form 
document, using a near-infrared light source, and 
also (in a separate, independent process) captures 
visually scanned images of unmarked form docu- 
ments, using a visible spectrum light source, and 



stores both. When display of a response-marked 
form is desired, the two types of images stored are 
merged to create combined images. 

In the preferred embodiment, one of the im- 

5 ages captured is a "blank" or unmarked master 
form. Far from being blank visually, this form is 
only "NIR blank" and typically contains rules, lines, 
boxes, preprinted text, logos, graphics or other 
identifying characteristics which do not change 

10 from user to user (e.g., respondents filling out 
forms). The unmarked master form image is cap- 
tured using a VIS scanner and stored as an unmar- 
ked master document image in a data processing 
system. 

75 Additional images are captured from response- 
marked copies of the master form containing data 
filled in by a respondent using a No. 2 pencil or 
other marking instrument producing detectable 
marks. Information on these documents is scanned 

20 using an OMR scanning system with a special 
light-detector system responsivity (i.e., principally 
sensitive in the NIR range). Due to the special 
spectral responsivity, the preprinted matter on the 
response-marked master form is not detected with 

25 the OMR scanner; only the variable response data 
marks are captured and stored as unique response 
record information in the data processing system. 

When an image of a specific response-marked 
document is needed, the data processing system 

30 merges the unmarked master form image with the 
unique response record information image to pro- 
duce a single document that can be viewed either 
on a screen or in hard copy. 

A primary objective of the present invention is 

35 to provide a method and apparatus that allows a 
VIS image of an unmarked master form to be 
combined or merged with an OMR image of a 
response-marked form to produce a single image 
that is comparable in quality and appearance to a 

40 conventionally-captured VIS image. 

Another objective is to reduce the amount of 
memory space required to store the information 
associated with response-marked form documents 
with variable recorded data. 

45 A further objective is to provide a method to 

reproduce the entire contents of a scanned, 
response-marked form; to allow display, interpreta- 
tion or analysis of any of the discrete information 
recorded on that form; and to reduce the amount of 

50 computer processing power required to accomplish 
this objective. 

A still further objective is to provide an image 
processing system that permits scanned images to 
be merged and to incorporate the merge technique 

55 successfully in "real world" documents that contain 
normal printing tolerances, dimensional variations 
due to temperature and humidity, mis-registration 
and feeding artifacts that occur during document 
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image-capture and other anomalies that occur at 
various points in the scanning process. 

These and other objectives of the present in- 
vention will become apparent with reference to the 
drawings, the detailed description of the preferred 
embodiment and the appended claims. 

DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of an apparatus for 
capturing, storing and merging multiple scanned 
images in accordance with the present invention. 

Figure 2a is a plan view of a simplified, unmar- 
ked master form document as used in the present 
invention. 

Figure 2b is a plan view of a copy of a simpli- 
fied master form document as in Figure 2a on 
which response data have been recorded. 

Figure 3 is a flowchart showing how an unmar- 
ked master form document and multiple response- 
marked form documents are scanned and stored in 
accordance with the present invention. 

Figure 4 is a flowchart showing how two cap- 
tured images are merged and displayed in accor- 
dance with the present invention. 

Figure 5a is an image of a simplified, unmar- 
ked master form document as stored by the 
present invention. 

Figure 5b is a plan view of the simplified mas- 
ter form document of Figure 5a, now with re- 
sponses marked, and also a view of the merged 
image produced by merging the images of Figures 
5a and 5c. 

Figure 5c is an image of the response-marked 
form of Figure 5b as stored by the present inven- 
tion, with only the response marks and triangulation 
marks captured. 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

Referring to Figures 1, 2a and 2b, the basic 
operative components of an apparatus used to cap- 
ture, store and merge scanned images in accor- 
dance with the present invention are shown. 

While, as will be described below, the present 
invention is applicable to many data handling tasks, 
for purposes of explanation, a task involving data 
collected by having respondents mark on a stan- 
dard form document will be described. Figure 2a 
shows a simplified, unmarked master form docu- 
ment 100 with six response areas of the kind 
typically used on scanned documents. These con- 
sist of three multiple choice response areas 110 
(Nos. 1-3), one yes-no response area 111 (No. 4), 
one true-false response area 112 (No. 5) and one 
fill-in-the-blank area 113 (No. 6). The circles and 
underscores define preprinted response areas con- 



taining indicia such as A, B, C or Y and N or T and 
F. These indicia correspond to response options for 
a predefined set of questions or other response- 
stimulating items, usually found in a printed booklet 

5 with which the respondent works but which may 
also be on the master form 100 itself. All notations 
on the document in Figure 2a are preprinted. Fig- 
ure 2b shows the same form as in Figure 2b, 
except that responses have now been recorded, by 

70 marking the defined response areas or filling in 
blanks. 

In a typical data collection application unmar- 
ked form documents will be distributed to dozens, 
hundreds or thousands of respondents. Each re- 

15 spondent is asked to record his or her responses 
by marking on the form provided. When the 
response-marked forms are collected, the usual 
task is to scan each one to extract the response 
data and convert it to more easily manipulated 

20 electronic data form. While the original response- 
marked forms may thereafter be discarded, in the 
applications of interest for the present invention, 
the user desires to maintain the ability to archivally 
store and later recover an image of the actual 

25 response-marked forms. The present invention pro- 
vides a method and apparatus for storing electron- 
ically only the variable data from the response- 
marked forms and recovering the actual image by 
merging such stored images of variable data with a 

30 stored image of the unmarked form master. 

As shown in Fig. 1 , the unmarked master form 
document 100 is scanned using a scanner 18 with 
visible spectrum illumination (VIS mode) to capture 
all visual information on the scanned side of the 

35 document, such as preprinted text, lines and re- 
sponse areas outlines. This information is stored on 
memory device 14 as a master form image record 
32. Because most forms contain a certain amount 
of unprinted space, a standard "lossless" image 

40 compression program (such as those known under 
the names "Runlength", Huffman or "LZW") is 
preferably used to process and size-reduce the raw 
pixel data in the frame grabber within scanner 20 
before "permanently" storing the master form im- 

45 age record 32. Thus, the image stored accounts for 
every pixel in the image of the master form 100, 
but "lossless" compression permits this to occur in 
something less than the storage required for a non- 
compressed, full, pixei-by-pixel image. 

50 Response-marked documents 101 are scanned 

in an OMR scanner 20, preferably one using an 
NIR mode. (It would be possible to avoid the use of 
two separate scanners 18, 20 with a single scanner 
capable of scanning in both VIS and NIR modes. 

55 For example, by switching illumination sources or 
by using a filter that only passes near-infrared light, 
the scanner 18 could be changed to a mode in 
which it will detect only response markings made 
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with pigments containing carbon or graphite that 
absorbs this light spectrum. Another approach 
would be to employ a single scanner with illumina- 
tion covering both NIR and VIS light ranges, then 
using beam splitting and filters to carry out NIR 
and VIS mode scanning simultaneously. The basic 
construction for a scanner suitable for use in the 
present invention is shown in U.S. Patent No. 
4,300,123 issued November 10, 1981, naming John 
V. McMillin and Dale W. Schroeder as inventors.) 
The NIR or OMR mode scanner 20 scans each 
response-marked form document 101, developing 
in the frame grabber within scanner 20 a complete 
pixel image of response markings on that docu- 
ment. Again, a conventional lossless program (such 
as those identified above) is preferably used to 
store the image. Because the preprinted matter on 
the document 101 is not detected in this OMR 
mode scan, the response image record wili only 
consist of spatial (x,y) location and grayscale in- 
formation for a relatively small number of areas on 
the document 101. Accordingly, a very high degree 
of compression can be obtained, over 90% in a 
typical application. In effect, using a scanning 
mode that does not detect the preprinted portions 
of the response-marked form document results in a 
substantial "pre-compression" of the data to be 
stored, before the conventional compression step is 
performed. This doubly-compressed information is 
stored on memory device 14 as unique response 
image record 38. If multiple response-marked doc- 
uments 101 are scanned, the process of OMR 
scanning and storing additional unique response 
image records 38 Is repeated with subsequent 
response-marked documents. 

A computer program in the data processing 
device 12 directs the scanning process and storage 
of records 32,38, invoking the compression pro- 
grams as needed to process full pixel images of 
scanned documents. If desired, the amount of data 
captured and stored in either VIS or NIR mode can 
be reduced by defining certain areas of interest as 
the sole areas for which image data will be pre- 
served. For example, in the forms 100, 101 in 
Figures 2a and 2b, item No. 6 and the associated 
response blanks might be defined as an area of 
interest. 

Figure 3 shows a simplified flowchart of the 
scanning and record storage process. It includes 
the steps of scanning 31 the unmarked master 
form document in VIS mode to fill the frame grab- 
ber, compressing 32 the pixel image in the frame 
grabber, storing 33 the compressed master form 
image record, scanning 35 a response-marked 
form document in OMR/NIR mode, compressing 36 
the pixel image in the frame grabber, storing 37 the 
compressed response markings only as a unique 
response image record, determining 40 whether 



there are more response-marked documents to 
scan and either returning 42 to the response- 
marked document scan or transferring 44 control to 
a wait mode, waiting 52 for an image retrieval 
5 request (see Figure 4). 

Even if no display of the response-marked doc- 
uments is ever required, the present invention pro- 
vides a significant advance over prior art systems 
for capturing data on form documents, because of 

10 the dramatic reduction in the data that needs to be 
stored. Moreover, by using two spectrally distinct 
scanning modes, including an NIR mode for detect- 
ing response markings, the present invention 
avoids a difficulty that is present in performing 

is OMR reading in a VIS mode. In a VIS mode, all the 
preprinted ink on a response marked form acts as 
noise that makes the detection of markings more 
difficult Further advantages of the invention arise 
by reason of its capability for reproducing the im- 

20 age of any response-marked document that has 
been scanned and saved in accordance with the 
preceding methods. This is done by merging the 
VIS-mode captured and stored image of the un- 
marked master form document with the desired 

25 NIR-mode captured and stored image of markings 
extracted from a response- marked form. 

Referring to Figures 1 and 4, the basic oper- 
ative components and steps used in displaying a 
merged image are shown. When an image of a 

30 particular response-marked form document 101 is 
requested, both the master form image record 32 
and the unique response image record 38 for the 
requested, response-marked form are retrieved 54, 
56 from storage 14. These images are merged in 

35 the data processing system 10 in the manner de- 
scribed below and the merged image is displayed 
60 on a display 16 (such as a CRT or flat panel 
electronic display) or printed on a laser or other 
printer 22. If desired, the resulting merged image 

40 may be stored 60 for archival purpose on archival 
storage medium 15. In general, the merger process 
simply consists of decompressing the respective 
stored images and (AOI images if merger of a 
defined pixel image area less than a full document 

45 is desired) overlaying the grayscale value for each 
pixel representing response markings from a re- 
sponse image record 38 on the corresponding pixel 
(same location) from a master form image record 
32. Grayscale values may vary from the simplest 

so situation where there are only two levels in the 
grayscale range, black or white, represented by a 
single bit, to grayscale ranges having 256 or more 
possible levels represented by eight or more bits. 
That is, the grayscale value for a pixel from re- 

55 sponse image record 38 simply replaces the cor- 
responding grayscale value from master form im- 
age record 32. (Variations on this replacement ap- 
proach are discussed below.) 
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Either at the time the records 32, 38 are stored 
or at some later time before they are merged, the 
records must undergo one or more steps of pre- 
processing, discussed below, that aid or enhance 
the merger process. For simplicity, Figure 4 shows 
a single preprocessing step 57 occurring before 
the step 58 of merging two images. Such pre- 
processing will usually include the correction of 
skewing or other anomalies known to occur during 
scanning that impair the integrity of scanned im- 
ages. 

The simplest case of merging two images cap- 
tured by the present invention is illustrated in Fig- 
ures 5a, 5b and 5c. Figure 5a shows an image of a 
highly-simplified, unmarked master form document 
70a having two response items 71, 72, each with 
an A and a B response option, as indicated by the 
labels within circular response areas or bubbles 
71a, 71b, 72a, 72b. Form image 70a also has three 
triangulation marks 90, 92, 94, used for alignment, 
as will be explained below. It should be noted that 
the triangulation marks 90, 92, 94 are preprinted 
but are printed in ink (e.g., black ink) that will be 
detected in the NIR mode scan as well as in the 
VIS mode. Form 70a is "blank" or unmarked; it has 
no recorded responses and no spurious marks. 
Figure 5b shows a response-marked copy 70b of 
master form 70a on which response marks 80 and 
82 have been placed by the respondent. The trian- 
gulation marks 90, 92, 94 also appear here. Figure 
70c shows the image of Figure 70b that is obtained 
by an OMR scan of marked copy 70b. Only the 
two marks 80, 82 and the triangulation marks 90, 
92, 94 are part of this image. The triangulation 
marks 90, 92, 94 are the only part of this image not 
marked in by the respondent. 

If perfect form printing and scanning are as- 
sumed and if all response-marked forms 70b are 
the same size as and are scanned at the same 
dots per inch (DPI) resolution and grayscale range 
as the unmarked master form image 70a, then both 
the master form image record 32 and the unique 
response image record 38 that represents Figure 
5c will have the same bit-map image formats after 
decompression into the system's merged storage 
buffer (temporary storage) prior to CRT viewing, 
hard copy printing or archiving of the merged im- 
age. That is, there will exist a perfectly aligned, 
one-to-one correspondence between each pixel lo- 
cation in these respective stored images and their 
grayscale information will correspond directly. In 
this circumstance, there is no need to do any 
dimensional scaling, and the triangulation marks 
90, 92, 94 are unnecessary. To perform a merge, 
all that is needed is to know the x-y offsets of the 
pixel information representing the two marks 80, 82 
from some standard origin point and to decide on 
the rule for overlaying unique response image 



record 38 on master form image record 32. If the 
overlay rule is that the grayscale value for each 
pixel in each unique response image record 38 
replaces the corresponding grayscale value for the 

5 same pixel in the master form document record 32, 
then the merger of the two images is complete as 
soon as this replace operation is carried out by the 
data processing system 12 for each pixel stored as 
part of unique response image record 38. If this 

w merged document is displayed on CRT 16 or print- 
ed on laser printer 22, it will appear to be the same 
as the image of Figure 5b, showing the marked 
master form 70b. 

While some real-world document scanning and 

15 merging tasks may approach the simplicity of the 
example just described, it is likely that the require- 
ments of the particular scanning and merger ap- 
plication of interest will be more challenging and 
require various forms of additional processing of 

20 the images merged, either at the time of image 
capture or the time of merger (or both) to provide 
the desired image. The following is an itemization 
of variations on the simplest merge task that may 
be encountered and may be accommodated by the 

25 present invention: 

1. DPI scaling needs, caused by images of 
different dimensions and or with different DPI 
scales; 

2. grayscale differences or grayscale adjust- 
so ments; 

3. variations in merge operators; 

4. changing of location offsets of merged data; 

5. correction of scanner or printing anomalies, 
including, rotation or skew problems, non-linear 

35 rotation and optical distortion. 

DPI Scaling 

In the preferred embodiment of the present 

40 invention, unmarked master form documents cap- 
tured using VIS mode may be scanned at a dif- 
ferent pixel dimension resolution than the 
response-marked forms scanned using OMR 
mode. Through the use of a scaling program con- 

45 tained in the data processing system 12, the differ- 
ences between these two pixel dimension resolu- 
tions can be adjusted and the merged document 
can be displayed in the desired resolution. 

For example, suppose unmarked master form 

50 image 70a is captured at 120 DPI horizontally and 
90 DPI vertically, and that response-marked form 
image 70b is captured at 200 DPI horizontally and 
150 DPI vertically. Furthermore, suppose both im- 
ages represent an 8.5" x 11" page-size source 

55 document. Accordingly, the pixel dimensions of 
unmarked master form image 70a are 1020 x , 990 y , 
whereas response-marked form image 70b has 
pixel dimensions of 1700 x , 1650 y . Let the scaling 
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ratio factors for image 70a be set at 2.5 X , 3.33 y> 
and the scaling ratio factors for image 70b be set 
at 1 .5 X , 2.0 y . Thus the new, rescaled pixel dimen- 
sions for both of the images are now 2550 x , 3300 y , 
or equivalent to synthesized resolution of 300 DPI. 
For this set of scaling ratios, either of the two 
separate images, or the rescaled merged image 
will appear in the hard-copy in a 1:1 ratio to "true 
size" when printed on a 300 DPI laser printer. A 
variety of scaling ratio sets can be selected to 
uniformly decrease or increase the final hard copy 
size or the CRT display size with respect to the 
original dimensions and the DPI resolution of the 
printer. The scaling program preferably incorpo- 
rates interpolation ramps rather than fat-bit replica- 
tion, which provides smoothed grayscale gradients 
when upscaling or, conversely, generates optimum 
pixel subsampled bit-maps when downscaling. 

Grayscale Differences 

In accordance with this invention, an unmarked 
master form document that is scanned at a dif- 
ferent pixel intensity range (or grayscale content) 
than a response-marked form is remapped (or vice 
versa) to adjust for these differences, before the 
images are merged by a program contained in the 
data processing system 12. 

This is accomplished by remapping one or 
more of the images into a common level of 
grayscale. Using the remapping program, the one- 
time scan of the unmarked master form may be 
performed at high grayscale content and high DPI 
resolution and the higher speed, higher volume 
batch run of data-bearing forms may be captured 
at lower grayscale content and DPI resolution. 
Benefits from this preferred method include, in- 
creasing processing speed, compression efficiency 
and meeting space storage limitations. 

Any number of bits/pixel from 1 to 8 may be 
used for the grayscale. In most practical document 
image-capture systems, the grayscale range is 
from 2 levels up to 256 levels, or an equivalent 
range of 1 to 8 bits for each pixel of the image. For 
example, if the unmarked master form is scanned 
(VIS-mode) at 8 bits/pixel for maximum quality 
(e.g., the dynamic index range is 000-255), but the 
data-bearing form is scan-captured at only 4 
bits/pixel (OMR-mode), the latter must be re- 
mapped in such a way that the 16 grayscale levels 
(4-bits/pixel) span the entire 000-255 range of the 
"master" before the merging step takes place. 
Thus, the grayscale Hex 1 level for the 4-bit image 
is linearly re-mapped to index 17, grayscale Hex 2 
level is remapped to index 34, and so on, with 
grayscale Hex 14 remapped to 238, and grayscale 
Hex 15 to 255. Non-linear re-mapping (e.g., loga- 
rithmic), or gamma control processes, such as are 



common to manipulate contrast levels in the photo- 
graphic industry, can be readily implemented for 
even greater flexibility and control in the merge 
sequence. 

5 Another point on grayscale levels that affects 

image quality can be mentioned. When response- 
marked document images are captured in OMR 
mode, the white background index level should be 
set slightly closer to saturation-white level (index 

w 255 in an eight-bit per pixel system) than the white 
level for VIS mode scanning of the unmarked mas- 
ter form. This assures grayscale-level uniformity in 
the background of the merged document because 
a merging program selects the darker of the two 

15 pixel values when the images are merged (in the 
absence of a different merge operator). 

Merge Operators 

20 In the normal use of the merge technique, the 

merging of marked response data onto a corre- 
sponding master form implies that this step be 
performed mathematically in the "merge" Sense; 
that is, the darker index value of each pixel 

25 (furthest away from white 255 or closest to 000) is 
the surviving pixel value in any merged area. How- 
ever, a fully flexible merge algorithm can offer an 
arbitrary "merge operator" for the output pixel from 
any two (or more) sets of bit-map images; that is, 

30 the pixel resultant from merger of differing 
grayscale values can be opaquely joined (e.g., re- 
place the old with the new value regardless of 
respective index values) or could be cumulative, or 
for that matter, follow any other pre-defined Bool- 

35 ean or algebraic expression. 

Location Offsets 

Any one image to be merged onto another 

40 may be offset by an arbitrary magnitude from a 
"master" reference set of x,y coordinates, instead 
of simply being merged by direct overlay onto the 
same pixel area from which it originates. The ability 
to mathematically "slide" one bit-map image ar- 

45 ound over the "surface" of another before the 
actual merging process is executed is useful for 
the successful output of hard-copy or for a CRT 
display. Otherwise, the user will often observe a 
misalignment of the (merged) "real world" variable 

so data from a response-marked form as it is placed 
into the preprinted context of the unmarked master 
form. Such misalignment, if sufficient, may cause 
the "authenticity" of the merged output image to 
be questioned. Therefore, all forms of the set to be 

55 included within the "merged composite" family 
must contain three or more pre-printed indicia or 
"triangulation marks" (detectable in each of the 
respective scanning spectrums, such as VIS, NIR, 
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etc.) so that the "sliding" or translation algorithm 
"knows" the x,y pixel offset values from "home 
position" for each image, and thereby knows how 
much to "slide" any or all of the individual bit-maps 
before the actual merge is executed. It is important 
to note, that in the general case, it is not required 
that the three triangulation marks be in the same 
location on the merged forms, but only that the x,y 
reference coordinates be known, a priori, (e.g., 
preprinted ID codes on the sheet) for the respec- 
tive document images, before merging takes place. 
When a "full-size" image is being merged onto 
another there must necessarily be a "cropping 
effect"-that is, loss of boundary/edge data, when- 
ever the translation offsets have finite values (e.g., 
non-zero ± N pixels). In some applications, the 
image that is translated to another location before 
merger may be an image of only a small area of 
interest. For such a merge, cropping is rarely a 
problem, because the area of interest that is trans- 
lated will have been selected and moved to a 
particular location for a purpose and with due con- 
sideration to the size of the image translated and 
the location of the area onto which it will be over- 
laid. 

Correction of Anomalies 

Occasionally a document scanned (either an 
unmarked master form or a response-marked form) 
is not fed perpendicularly to the scanning axis, but 
rather is skewed or rotated slightly during the im- 
age capture process. Even if all other geometric 
parameters and scaling factors are properly ac- 
counted for, the merged image will exhibit incon- 
gruence between the separate components that are 
merged if skew is present. Therefore, by means of 
detecting the presence and interpreting the x,y 
location of three or more triangulation marks from 
each scanned image, a rotating program in the 
data processing system 12 can deskew and correct 
any rotated images before the actual merge se- 
quence is executed. As mentioned earlier, the set 
of triangulation marks need not be in the same 
location on all documents, but merely their location 
pre-defined for each document type in the set. 
Moreover, the triangulation marks permit correction 
of rotation anomalies, whether they are from clock- 
wise or counterclockwise rotational skew errors. 

As discussed above, the two images to be 
merged may have different DPI resolutions, sizes, 
pixel X( y dimensions, aspect ratios, or spectral re- 
sponsivities. This further implies that the two docu- 
ments may be captured on different scanners. For 
example, the unmarked form master, which re- 
quires a one-time scan, may be captured on a low 
speed (a minute or more per capture), inexpensive, 
but high resolution (300 DPI) VIS-mode desktop 



scanner, whereas the response-marked forms are 
likely to be scanned on a high-performance OMR- 
mode scanner at thousands of documents/hour. 
The respective optical front end path length of 

5 these two scanners comprised of lens, mirrors, 
glass plates, light source, CCD array length and 
pitch, magnification factors, F/stop settings, focal 
length, and bandwidth block/pass filters are nec- 
essarily different. 

10 All optical systems comprised of the preceding 
components exhibit various distortions or nonlinear- 
ity along the 8.5" width of a full-size page docu- 
ment. For example, if one edge of the sheet is 
defined as the 0-pixel reference, then for a 1020 

rs pixel length along the 8.5" scan-axis (e.g., a 120 
DPI resolution), the quarter-distance point of 2.125" 
(8.5/4 = 2.125) ought to occur at pixel No. 255 
(1020/4 = 255) in the CCD (charge coupled device) 
array used to detect scanned images in most ad- 

20 vanced OMR scanners. Actual measurements, 
however, will show a several pixel departure from 
this ideal, and the error will vary from system to 
system as the above parameters are changed. The 
focal length and optical quality of the lens, in 

25 particular, have a significant influence on X-axis 
scan nonlinearity. 

Earlier discussions explained how two images 
with different DPI's, sizes, x-y offsets, rotations and 
pixel dimensions can be congruently aligned with 

30 the merge technique discussed. This capability, as 
described in the programs included in the preferred 
embodiment, assumes that the various image pro- 
cessing operations are dimensionally "linear" or 
isomorphic with respect to the underlying math- 

35 ematical rules. For example, an x-scaling ratio of 
2.5 is applied to all x-sweeps of the entire image, 
or the entire image is rotated 0.31 ° , or the entire 
image is "slid" +5 X ,-7 V pixels, and so on. The x- 
axis optical nonlinearity is corrected using a scaling 

40 program to apply a "transform function or lookup 
table" against the nominal scaling ratio. In the cited 
example, the 2.5X ratio must be further modulated 
by some function (a 5th order polynomial is typi- 
cally used to specify lens distortion) to correct for 

45 the discrepancy caused by lens distortion (and 
minor shifts caused by the reading head glass 
plate diffraction offsets) between the pixel number 
and actual detected position on the document. The 
function (or lookup table) itself is heuristically de- 

50 termined by measurements taken from precision 
optical targets (e.g. black/white line-pairs printed on 
a stable, white-reflecting base, such as Kodak Es- 
tar). The line-pair frequency is made high enough 
to identify individual pixel x-axis locations and 

55 shifts when viewing the CCD output signal on a 
scope, or analyzing corresponding pixel index val- 
ues from the digital bit-map of the frame-grabbed 
optical target. 
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Once the offset-error coefficients are quanti- 
fied, another program adjusts the 
expander/contractor scale-ratio factor on a point by 
point basis (e.g., pixel by pixel across the array 
length) to include the correction for the optical 
nonlinearity. Thus for the cited 2.5X scaling exam- 
ple, the pixel indices are "wobbled" by small ±A f s 
for the new bit-map index values. Finally, in the 
general-case solution, each image-if any two im- 
ages to be merged were captured under different 
optical path parameters-is corrected during scaling 
to the common merged image with the appropriate 
correction function (table) to avoid or minimize in- 
congruence in the final output. 

In general, documents improperly fed through 
a scanner exhibit an error comprised of rotation 
and xy-trans!ation, with the magnitude of the rela- 
tive components changing in some nonlinear (but 
continuous) fashion throughout the length of the 
document This point is explained by the following 
example. If the left-side lead-edge of an 11" long 
document arrives at the scan axis one-fourth inch 
earlier than the right side, the entry skew angle is 
1.3° (from elementary trigonometry, A s = sin- 1 
(1/4 H /11") = 1.3 degrees). Additionally, assume 
that checking the preprinted triangulation points 
shows that the document is "slid" sideways 
(outward) by 0.1"; that is, an x-axis translation 
offset error is present as well as the rotational 
error. Further suppose that when the trail-edge of 
the document leaves the scan axis that the left and 
right sides of an 8.5" wide document are aligned, 
e.g., the departure skew is 0° , and finally, the 11" 
registration edge of the document is tightly against 
the registration plate of the scanner. This implies 
that the translation error has also subsided to 0. 
Thus both components of registration error (rotation 
and translation) have changed continuously 
(following some unknown nonlinear function) during 
the transit of the sheet through the X-scan axis. To 
correct the stored image of a mis-registered docu- 
ment, adequate coordinate reference information 
from accessible triangulation marks is required to 
dynamically update the scaling, rotation and trans- 
lation correction programs as described in the 
present invention. Technically, the task is defined 
as "two dimensional unwarping". In a strict sense 
triangulation implies only three triangulation marks, 
as is well known from mensuration theory. But 
severe nonlinearities (as described above) may re- 
quire a multiplicity of triangulation marks well be- 
yond the basic three. For example, the regularly 
spaced NIR or VIS detectable preprinted timing 
track marks found on many OMR forms could 
serve as an additional source of triangulation data. 
In particular, applications of the present invention, 
processing of stored image data to make such 
corrections would be desirable. 



Additional Merge Applications 

While the most common use of the present 
invention will be merger of a single image (or 
s portion of an image) representing a response- 
marked form document onto the image of an un- 
marked master form document, the present system 
permits other merger operations to be performed 
as well. 

10 NIR-mode images from multiple response- 

marked forms can be merged into a common un- 
marked master form scanned in VIS mode. For 
example, a student is asked on an exam to draw 
his/her best representation of a "mathematically 

75 correct" ellipse about a preprinted cross-hair point, 
with the major and minor axes having a two to one 
ratio, and a maximum dimension of 12 centimeters. 
In this case, the master form may contain preprin- 
ted gridlines, the crosshair, or other nomenclature. 

20 The students' sheets contain only the crosshair. 
Based upon this approach, any number of NIR 
mode images of student responses can be super- 
imposed on a hard-copy printout of the master 
form; the effect being a "topological density" chart 

25 giving an easy-to-interpret visual presentation of 
the paths followed by the group of respondents in 
their collective efforts to draw the requested ellipse. 
Numerous other application examples can be read- 
ily imaged, such as the overlay of two specimen 

30 signatures, so as to note similarities or differences 
in size and style and possible indication of forgery, 
fraud or other misrepresentation. With wider 
grayscale ranges and suitable merge operators, 
these techniques may enhance photographic over- 

35 lay work for forensic analysis. 

Opaque merging (pasting) where "last-in" se- 
quence is preserved may be used for such multiple 
image overlays. In this case, an image from one or 
more successive documents is pasted into the 

40 master form in a Boolean opaque manner, such 
that the last image merged into the master form is 
"on top of" any earlier pasted in images. To the 
extent that the last image is smaller or offset from 
the earlier images, the outlying fringes of preceding 

45 image pasted-transfers will show in the composite. 

Using cut, paste and merge programs, multiple 
areas of interest (AOI's) can be cut from any of the 
captured images and pasted onto the merged out- 
put. A common application using multiple merges 

so is where a multiple set of small rectangular arrays 
are captured in the OMR mode from a given docu- 
ment (rather than the entire marked response im- 
age on the document), then merged into the mas- 
ter form at the respective coordinate locations from 

55 where each AOI window or segment originated 
(after any translation/skew/scaling operations have 
been executed). These programs can process (e.g., 
extract) one or more sets of origin x,y coordinate 
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values or alternatively, accept windowed images 
with coordinate origin header values, and place 
these AOPs into the specified locations on the 
merged output 

The same spectrum scan (either VIS or NIR) 
may be utilized on two or more merged images. 
For example, several scan passes may be made of 
the same unmarked master form, but under dif- 
ferent lighting conditions or lens settings or other 
optical conditions, or when using different printed 
samples from a press run to gauge accuracy and 
consistency of printed product. An examination of 
the merged composite, either in hard-copy output 
or on a CRT display, will quickly reveal the quality 
and repeatability of the master sets. Secondly, 
multiple NIR-mode user forms, with a preprinted 
(filled in) response mark pattern, can be scanned 
and the resulting images merged into a common 
output image to check tolerances and registration 
limits on the scanner feed mechanism in the scan- 
ner in which they were processed. 

The NIR mode and VIS mode scanned docu- 
ments may be selected to have different preprinted 
formats. For example, the unmarked "master form" 
scanned in VIS mode may contain a series of 
row/column boxes (roster layout) to show a group 
of names, signatures, test ID numbers, and re- 
sponses captured from a batch run of many sepa- 
rate NIR mode documents, whereas the NIR mode 
documents have only one "box" entry per sheet for 
each respondent's use. Alter batch scanning of the 
NIR mode documents is completed, the individual 
image segments or AOI's extracted from each NIR 
mode document are merged onto the common 
"master form", with the merge algorithm calculat- 
ing appropriate x,y offsets to place the successive 
AOI's in a roster sequence on the "master form". 

Although the description of the preferred em- 
bodiment has been presented, it is contemplated 
that various changes could be made without deviat- 
ing from the spirit of the present invention. Accord- 
ingly, it is intended that the scope of the present 
invention be dictated by the appended claims rath- 
er than by the description of the preferred embodi- 
ment 

Claims 

1. A method for storing multiple scanned images 
of a response-marked form document, com- 
prising the steps of: 

(a) scanning a copy of the form document 
on which no response information has been 
marked, using a scanning method that de- 
tects substantially all visual information 
present on the form document and pro- 
duces a pixel image thereof; 

(b) storing the pixel image data from step 



(a) in a computer storage device; 
(c) scanning at least one copy of the form 
document that has been marked with at 
least one response mark that absorbs light 

5 falling outside the visible spectrum, using a 

scanning method that detects only the pres- 
ence of light outside the visible spectrum 
and produces a pixel image of said at least 
one response mark; 

w (d) compressing the pixel image produced 

in step (c) to reduce the amount of pixel 
data used to represent areas of the form 
document bearing no response marks; and 

(e) storing the pixel image data obtained 
15 from step (d) in a computer storage device. 

2. The method recited in claim 1 further compris- 
ing the steps of: 

(f) for each response-marked copy of the 
20 form document for which a reproduction of 

the document is desired, decompressing 
the pixel data stored in step (e) to obtain 
the pixel image of step (c), and performing 
a pixel for pixel merge of the pixel images 
25 of steps (a) and (c) to produce a single 

merged pixel image; and 

(g) outputting the merged pixel image from 
the computer processing system. 

30 3. The method of claim 2 wherein step (g) com- 
prises printing the merged image or displaying 
the merged image on a video display appara- 
tus. 

35 4. The method of claim 1 wherein the scanning of 
the document in step (a) is done at a different 
pixel dimension resolution than the scanning of 
the document in step (c) and further compris- 
ing adjusting the pixel images of steps (a) and 

40 (c) with at least one scaling factor to resolve 

the differences between the two pixel dimen- 
sion resolutions. 

5, The method of claim 1 wherein the scanning of 
45 the document in step (a) is done at a different 

pixel grayscale range than the scanning of the 
document in step (c) and further comprising 
adjusting at least one of the pixel images of 
step (a) or (c) in a remapping process to 
so resolve the differences between the two pixel 

grayscale ranges. 

6. The method of claim 2 wherein the documents 
scanned in steps (a) and (c) contain at least 

55 three triangulation marks that are detectable by 

both the visual scanning of step (a) and the 
nonvisual scanning of step (c) such that the 
computer processing system can use the trian- 
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gulation marks in step (f) to properly align the 
two pixel images. 

7. The method of claim 6 wherein the triangula- 
tion marks are used by the computer process- 
ing system to deskew the pixel image data 
from both step (a) and step (c). 

8. The method of claim 1 wherein step (b) stores 
the unmarked form document in terms of one 
or more predefined areas of interests. 

9. The method of claim 1 wherein step (e) stores 
the response-marked form document in terms 
of one or more predefined areas of interests. 

10. The method of claim 1 wherein step (b) further 
includes the step of processing the master 
form to a predefined grayscale index level to 
insure that blank information on the master 
form will prevail in a merge with overlapped 
blank information in the unique record informa- 
tion. 

11. The method of claim 2 wherein step (f) further 
includes the step of processing the pixel im- 
ages of steps (a) and (c) to correct for optical 
distortion in the scanning steps (a) or (c), re- 
spectively. 

12. The method of claim 2 wherein the pixel-for- 
pixel merge uses a merge operator that selects 
the pixel with the darkest grayscale level as 
the surviving pixel, or selects the pixel from 
the response-marked document as the surviv- 
ing pixel regardless of respective grayscale 
values. 

13. The method of claim 2 further comprising the 
step of storing the merged pixel image in a 
computer storage device. 

14. An apparatus for storing multiple scanned im- 
ages of a response-marked form document, 
comprising: 

first means for scanning a copy of the 
form document on which no response informa- 
tion has been marked using a scanning meth- 
od that detects substantially all visual informa- 
tion present on the form document and pro- 
duces a first pixel image thereof; 

second means for scanning at least one 
copy of the form document that has been 
marked with at least one response mark that 
absorbs light failing outside the visible spec- 
trum, using a scanning method that detects 
only the presence of light outside the visible 
spectrum and produces a second pixel image 



of said at least one response mark; 

means for compressing the second pixel 
image produced by said second scanning 
means to reduce the amount of pixel data used 
5 to represent areas of the form document bear- 

ing no response marks; and 

means for storing the pixel image data 
obtained from said first and second scanning 
means in a computer storage device. 

70 

15. The apparatus recited in claim 14 further com- 
prising: 

means for decompressing the stored pixel 
image data to obtain for each response- 

15 marked copy of the form document for which a 

reproduction of the document is desired, the 
second pixel image and performing a pixel-for- 
pixel merge of the first and second pixel im- 
ages to produce a single merged pixel image; 

20 and 

output means for outputting the merged 
pixel image from the computer processing sys- 
tem. 

25 16. The apparatus of claim 15 wherein the output 
means comprises a printer or a video. 

17. A method for storing multiple scanned images 
of a response-marked form document, com- 
30 prising the steps of: 

(a) scanning a copy of the form document 
on which no response information has been 
marked, using a scanning method that is 
responsive to light of a first spectral range 

35 and produces a pixel image of those por- 

tions of the form document that absorb light 
in such range; 

(b) storing the pixel image data from step 
(a) in a memory device; 

40 (c) scanning at least one copy of the form 

document that has been marked with at 
least one response mark that absorbs light 
falling outside said first spectral range, us- 
ing a scanning method that detects only the 

45 presence of light outside the first spectral 

range and produces a pixel image of said at 
least one response mark; 

(d) compressing the pixel image produced 
in step (c) to reduce the amount of pixel 

so data used to represent areas of the form 

document bearing no response marks; and 

(e) storing the pixel image data obtained 
from step (d) in a memory device. 

55 . 
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